Systems and methods for generating skin tone profiles

ABSTRACT

A computing device with a digital camera obtains a reference image depicting at least one reference color and calibrates parameters of the digital camera based on the at least one reference color. The computing device captures, by the digital camera, a digital image of an individual utilizing the calibrated parameters. The computing device defines a region of interest in a facial region of the individual depicted in the digital image captured by the digital camera. The computing device generates a skin tone profile for pixels within the region of interest and displays a predetermined makeup product recommendation based on the skin tone profile.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to, and the benefit of, U.S.Provisional Patent Application entitled, “Method and Apparatus of SkinTone Estimation,” having Ser. No. 62/681,174, filed on Jun. 6, 2018,which is incorporated by reference in its entirety.

TECHNICAL FIELD

The present disclosure generally relates to systems and methods forgenerating skin tone profiles of individuals depicted in digital images.

BACKGROUND

With the proliferation of smartphones, tablets, and other displaydevices, people have the ability to capture digital images virtually anytime where application programs have become popular on smartphones andother portable display devices for managing and editing captured digitalcontent. However, due to variations in the color temperature setting incameras, environmental lighting, and so on, it can be difficult toaccurately estimate attributes (e.g., skin tone) of the facial region ofan individual depicted in a digital image. Therefore, there is a needfor an improved system and method for estimating skin tone profiles.

SUMMARY

In accordance with one embodiment, a computing device with a digitalcamera obtains a reference image depicting at least one reference colorand calibrates parameters of the digital camera based on the at leastone reference color. The computing device captures, by the digitalcamera, a digital image of an individual utilizing the calibratedparameters. The computing device defines a region of interest in afacial region of the individual depicted in the digital image capturedby the digital camera. The computing device generates a skin toneprofile for pixels within the region of interest and displays apredetermined makeup product recommendation based on the skin toneprofile.

Another embodiment is a system that comprises a digital camera, a memorystoring instructions, and a processor coupled to the memory. Theprocessor is configured by the instructions to obtain a reference imagedepicting at least one reference color and calibrate parameters of thedigital camera based on the at least one reference color. The processoris further configured to capture, by the digital camera, a digital imageof an individual utilizing the calibrated parameters. The processor isfurther configured to define a region of interest in a facial region ofthe individual depicted in the digital image captured by the digitalcamera. The processor is further configured to generate a skin toneprofile for pixels within the region of interest and display apredetermined makeup product recommendation based on the skin toneprofile.

Another embodiment is a non-transitory computer-readable storage mediumstoring instructions to be implemented by a computing device having aprocessor, wherein the instructions, when executed by the processor,cause the computing device to obtain a reference image depicting atleast one reference color and calibrate parameters of a digital camerabased on the at least one reference color. The processor is furtherconfigured to capture, by the digital camera, a digital image of anindividual utilizing the calibrated parameters. The processor is furtherconfigured to define a region of interest in a facial region of theindividual depicted in the digital image captured by the digital camera.The processor is further configured to generate a skin tone profile forpixels within the region of interest and display a predetermined makeupproduct recommendation based on the skin tone profile.

Other systems, methods, features, and advantages of the presentdisclosure will be or become apparent to one with skill in the art uponexamination of the following drawings and detailed description. It isintended that all such additional systems, methods, features, andadvantages be included within this description, be within the scope ofthe present disclosure, and be protected by the accompanying claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of the disclosure can be better understood withreference to the following drawings. The components in the drawings arenot necessarily to scale, with emphasis instead being placed uponclearly illustrating the principles of the present disclosure. Moreover,in the drawings, like reference numerals designate corresponding partsthroughout the several views.

FIG. 1 is a block diagram of a computing device for generating skin toneprofiles in accordance with various embodiments of the presentdisclosure.

FIG. 2 is a schematic diagram of the computing device of FIG. 1 inaccordance with various embodiments of the present disclosure.

FIG. 3 is a top-level flowchart illustrating examples of functionalityimplemented as portions of the computing device of FIG. 1 for generatingskin tone profiles according to various embodiments of the presentdisclosure.

FIG. 4 illustrates a region of interest defined by the computing devicein FIG. 1 according to various embodiments of the present disclosure.

FIG. 5 illustrates a luminance histogram for pixels within the region ofinterest generated by the computing device in FIG. 1 according tovarious embodiments of the present disclosure.

DETAILED DESCRIPTION

Various embodiments are disclosed for accurately generating skin toneprofiles of individuals depicted in digital images. Accuratedetermination of skin tone profiles is important for such applicationsas performing virtual application of makeup effects, recommendingcompatible makeup products, and so on. A description of a system forgenerating skin tone profiles is now described followed by a discussionof the operation of the components within the system. FIG. 1 is a blockdiagram of a computing device 102 in which the techniques for generatingskin tone profiles disclosed herein may be implemented. The computingdevice 102 may be embodied as a computing device such as, but notlimited to, a smartphone, a tablet computing device, a laptop, and soon.

A profile generator 104 executes on a processor of the computing device102 and includes a reference color extractor 106, a calibration unit108, a camera interface 110, and a content analyzer 112. The referencecolor extractor 106 is configured to obtain a reference image depictingone or more reference colors where the reference image may be depictedon a white balance card, a banknote, or other source with a known colorscheme. The calibration unit 108 is configured to calibrate parametersof the digital camera based on the one or more reference colors.

The camera interface 110 is configured to cause a digital camera tocapture a digital image of an individual. As one of ordinary skill willappreciate, the digital image may be encoded in any of a number offormats including, but not limited to, JPEG (Joint Photographic ExpertsGroup) files, TIFF (Tagged Image File Format) files, PNG (PortableNetwork Graphics) files, GIF (Graphics Interchange Format) files, BMP(bitmap) files or any number of other digital formats. Alternatively,the digital image may be derived from a still image of a video encodedin formats including, but not limited to, Motion Picture Experts Group(MPEG)-1, MPEG-2, MPEG-4, H.264, Third Generation Partnership Project(3GPP), 3GPP-2, Standard-Definition Video (SD-Video), High-DefinitionVideo (HD-Video), Digital Versatile Disc (DVD) multimedia, Video CompactDisc (VCD) multimedia, High-Definition Digital Versatile Disc (HD-DVD)multimedia, Digital Television Video/High-definition Digital Television(DTV/HDTV) multimedia, Audio Video Interleave (AVI), Digital Video (DV),QuickTime (QT) file, Windows Media Video (WMV), Advanced System Format(ASF), Real Media (RM), Flash Media (FLV), an MPEG Audio Layer III(MP3), an MPEG Audio Layer II (MP2), Waveform Audio Format (WAV),Windows Media Audio (WMA), 360 degree video, 3D scan model, or anynumber of other digital formats.

The content analyzer 112 is configured to define a region of interest ina facial region of the individual depicted in the digital image capturedby the digital camera. The content analyzer 112 is further configured togenerate a skin color profile for pixels within the region of interest.The content analyzer 112 is further configured to obtain makeup productrecommendations 118 from a data store 116 based on the generated skincolor profile and display the makeup product recommendation 118 in auser interface to the user of the computing device 102.

FIG. 2 illustrates a schematic block diagram of the computing device 102in FIG. 1. The computing device 102 may be embodied in any one of a widevariety of wired and/or wireless computing devices, such as a desktopcomputer, portable computer, dedicated server computer, multiprocessorcomputing device, smart phone, tablet, and so forth. As shown in FIG. 2,the computing device 102 comprises memory 214, a processing device 202,a number of input/output interfaces 204, a network interface 206, adisplay 208, a peripheral interface 211, and mass storage 226, whereineach of these components are connected across a local data bus 210.

The processing device 202 may include any custom made or commerciallyavailable processor, a central processing unit (CPU) or an auxiliaryprocessor among several processors associated with the computing device102, a semiconductor based microprocessor (in the form of a microchip),a macroprocessor, one or more application specific integrated circuits(ASICs), a plurality of suitably configured digital logic gates, andother well known electrical configurations comprising discrete elementsboth individually and in various combinations to coordinate the overalloperation of the computing system.

The memory 214 may include any one of a combination of volatile memoryelements (e.g., random-access memory (RAM, such as DRAM, and SRAM,etc.)) and nonvolatile memory elements (e.g., ROM, hard drive, tape,CDROM, etc.). The memory 214 typically comprises a native operatingsystem 216, one or more native applications, emulation systems, oremulated applications for any of a variety of operating systems and/oremulated hardware platforms, emulated operating systems, etc. Forexample, the applications may include application specific softwarewhich may comprise some or all the components of the computing device102 depicted in FIG. 1. In accordance with such embodiments, thecomponents are stored in memory 214 and executed by the processingdevice 202, thereby causing the processing device 202 to perform theoperations/functions disclosed herein. One of ordinary skill in the artwill appreciate that the memory 214 can, and typically will, compriseother components which have been omitted for purposes of brevity. Forsome embodiments, the components in the computing device 102 may beimplemented by hardware and/or software.

Input/output interfaces 204 provide any number of interfaces for theinput and output of data. For example, where the computing device 102comprises a personal computer, these components may interface with oneor more user input/output interfaces 204, which may comprise a keyboardor a mouse, as shown in FIG. 2. The display 208 may comprise a computermonitor, a plasma screen for a PC, a liquid crystal display (LCD) on ahand held device, a touchscreen, or other display device.

In the context of this disclosure, a non-transitory computer-readablemedium stores programs for use by or in connection with an instructionexecution system, apparatus, or device. More specific examples of acomputer-readable medium may include by way of example and withoutlimitation: a portable computer diskette, a random access memory (RAM),a read-only memory (ROM), an erasable programmable read-only memory(EPROM, EEPROM, or Flash memory), and a portable compact disc read-onlymemory (CDROM) (optical).

Reference is made to FIG. 3, which is a flowchart 300 in accordance withvarious embodiments for generating skin tone profiles performed by thecomputing device 102 of FIG. 1. It is understood that the flowchart 300of FIG. 3 provides merely an example of the different types offunctional arrangements that may be employed to implement the operationof the various components of the computing device 102. As analternative, the flowchart 300 of FIG. 3 may be viewed as depicting anexample of steps of a method implemented in the computing device 102according to one or more embodiments.

Although the flowchart 300 of FIG. 3 shows a specific order ofexecution, it is understood that the order of execution may differ fromthat which is depicted. For example, the order of execution of two ormore blocks may be scrambled relative to the order shown. Also, two ormore blocks shown in succession in FIG. 3 may be executed concurrentlyor with partial concurrence. It is understood that all such variationsare within the scope of the present disclosure.

At block 310, the computing device 102 obtains a reference imagedepicting at least one reference color. For some embodiments, thereference image can depicted on such objects as a white balance card, acolor checker, a banknote, a credit card, photocopy paper, tissue paper,a mobile phone, or a non-glossy white object. For such embodiments, theobject depicting the reference image is located at a predefined distancefrom the digital camera.

At block 320, the computing device 102 calibrates parameters of thedigital camera based on the at least one reference color, where suchparameters may include white balance level, exposure compensation, gammacorrection, and so on. At block 330, the computing device 102 captures adigital image of an individual utilizing the calibrated parameters.

At block 340, the computing device 102 defines a region of interest in afacial region of the individual depicted in the digital image capturedby the digital camera. For some embodiments, the computing device 102defines the region of interest by determining a color distance betweenpixels in the facial region and one or more predetermined target skintones and designating pixels within a threshold color distance of theone or more predetermined target skin tones as part of the region ofinterest.

For some embodiments, the computing device 102 defines the region ofinterest by identifying locations of predetermined feature points withinthe facial region and defining a boundary of the region of interestbased on the locations of the predetermined feature points. To furtherillustrate, reference is made to FIG. 4, which illustrates definition ofa region of interest 404 according to various embodiments. In theexample shown, the computing device 102 analyzes the facial region 402of the individual and identifies various feature points (shown as dots).The computing device 102 generates a region of interest 404 based on thelocation of the feature points.

In accordance with some embodiments, the computing device 102 detectsthe locations of various target feature points, which may comprise, forexample, the eyes, nose, mouth, eyebrows, and so on. The target featurepoints may also include the overall facial contour of the user's face.The computing device 102 then defines a region of interest 404 based onthe location of the feature points. As shown, the computing device 102may be configured to define a boundary based on a series of paraboliccurves defined at or near various feature points. In the example shown,the region of interest comprises the cheek and nose regions of the user.That is, in some embodiments, the region of interest may be predefinedbased on specific target regions or features of the user (e.g., thecheek and nose regions) where the boundary is then defined based on theactual feature points detected on the user's face such that the regionof interest encompasses those target regions or features.

Referring back to FIG. 3, at block 350, the computing device 102generates a skin tone profile for pixels within the region of interest.For some embodiments, the computing device 102 generates the skin toneprofile by generating a luminance histogram for the pixels within theregion of interest, removing predetermined portions of the luminancehistogram to generate a target histogram portion, determining a dominantcolor value based on the target histogram portion, and generating theskin tone profile based on the determined dominant color value. Thedominant color value may be determined by such techniques as calculatinga mean of the target histogram, calculating a peak of the targethistogram, calculating a weighted average of the target histogram, or bycalculating a mean based on a mean-shift clustering algorithm of thetarget histogram.

The luminance histogram illustrates the distribution of pixel brightnessof the region of interest, where the pixel brightness is typicallycomputed using either the Y component in the YUV color space or the Lcomponent in the Lab (or CIELAB) color space. To further illustrate,reference is made to FIG. 5, which illustrates a luminance histogram 502for pixels within the region of interest according to variousembodiments. For some embodiments, the computing device 102 removespredetermined portions 504, 506 of the histogram to generate a targethistogram portion.

One predetermined portion 506 may comprise, for example, the upper 30%of the histogram 502 that corresponds to pixels that are lighter (e.g.,pixels that are part of a reflective portion). Specifically, the upperportion generally corresponds to the light reflection that occurs on theregion of interest. Another predetermined portion 504 may comprise, forexample, the lower 30% of the histogram 502 that corresponds to pixelsthat are darker (e.g., pixels that are part of a shadow region).Specifically, the lower portion generally corresponds to the shadoweffect that occurs on the region of interest.

An average color value is determined based on the remaining histogramportion 508 (i.e., the target histogram portion), and the skin toneprofile is generated based on the determined average color value. Byexcluding the predetermined upper and lower portions of the histogram502, the computing device 102 is able to reduce the impact of light andshadow components on the average color calculation, thereby providingmore accurate skin tone estimation.

For some embodiments, the computing device 102 generates the skin toneprofile by extracting an illumination layer and a reflectance layer fromthe pixels within the region of interest and generating the skin toneprofile based on the reflectance layer. For some embodiments, thecomputing device 102 generates the skin tone profile by converting adetected skin tone from a first color space to a second color spacebased on a predefined transformation matrix or by mapping a detectedskin tone from a first classification to a second classification basedon a predefined lookup table.

At block 360, the computing device 102 displays a predetermined makeupproduct recommendation based on the skin tone profile. For someembodiments, each product recommendation 118 in the data store 116(FIG. 1) also includes a target RGB value or range of target RGB valuesassociated with a target skin tone color. Note that instead of targetRGB value(s), a target YUV value or range of target YUV values could bestored for each product recommendation 118. Similarly, a target Labvalue or range of target Lab values could be stored for each productrecommendation 118. The computing device 102 obtains one or more productrecommendations 118 from the data store 116 by matching the estimatedskin tone profile with one or more target RGB/YUV/Lab value(s) ofcorresponding product recommendations 118. Thereafter, the process inFIG. 3 ends.

It should be emphasized that the above-described embodiments of thepresent disclosure are merely possible examples of implementations setforth for a clear understanding of the principles of the disclosure.Many variations and modifications may be made to the above-describedembodiment(s) without departing substantially from the spirit andprinciples of the disclosure. All such modifications and variations areintended to be included herein within the scope of this disclosure andprotected by the following claims.

At least the following is claimed:
 1. A method implemented in acomputing device having a digital camera, comprising: obtaining areference image depicting at least one reference color; calibratingparameters of the digital camera based on the at least one referencecolor; capturing, by the digital camera, a digital image of anindividual utilizing the calibrated parameters; defining a region ofinterest in a facial region of the individual depicted in the digitalimage captured by the digital camera; generating a skin tone profile forpixels within the region of interest; and displaying a predeterminedmakeup product recommendation based on the skin tone profile.
 2. Themethod of claim 1, wherein the reference image is depicted on one of thefollowing objects: a white balance card, a color checker, a banknote, acredit card, photocopy paper, tissue paper, a mobile phone, a non-glossywhite object.
 3. The method of claim 2, wherein the object depicting thereference image is located at a predefined distance from the digitalcamera.
 4. The method of claim 1, wherein the parameters of the digitalcamera comprise at least one of: white balance level; exposurecompensation; and gamma correction.
 5. The method of claim 1, whereindefining the region of interest in the facial region comprises:determining a color distance between pixels in the facial region and oneor more predetermined target skin tones; and designating pixels within athreshold color distance of the one or more predetermined target skintones as part of the region of interest.
 6. The method of claim 1,wherein defining the region of interest in the facial region comprises:identifying locations of predetermined feature points within the facialregion; and defining a boundary of the region of interest based on thelocations of the predetermined feature points.
 7. The method of claim 1,wherein generating the skin tone profile for the pixels within theregion of interest comprises: generating a luminance histogram for thepixels within the region of interest; removing predetermined portions ofthe luminance histogram to generate a target histogram portion;determining a dominant color value based on the target histogramportion; and generating the skin tone profile based on the determineddominant color value.
 8. The method of claim 7, wherein determining thedominant color value based on the target histogram portion comprises oneof: calculating a mean of the target histogram; calculating a peak ofthe target histogram; calculating a weighted average of the targethistogram; or calculating a mean based on a mean-shift clusteringalgorithm of the target histogram.
 9. The method of claim 1, whereingenerating the skin tone profile for the pixels within the region ofinterest comprises: extracting an illumination layer and a reflectancelayer from the pixels within the region of interest; and generating theskin tone profile based on the reflectance layer.
 10. The method ofclaim 1, wherein generating the skin tone profile for the pixels withinthe region of interest comprises one of: converting a detected skin tonefrom a first color space to a second color space based on a predefinedtransformation matrix; or mapping a detected skin tone from a firstclassification to a second classification based on a predefined lookuptable.
 11. A system, comprising: a digital camera; a memory storinginstructions; a processor coupled to the memory and configured by theinstructions to at least: obtain a reference image depicting at leastone reference color; calibrate parameters of the digital camera based onthe at least one reference color; capture, by the digital camera, adigital image of an individual utilizing the calibrated parameters;define a region of interest in a facial region of the individualdepicted in the digital image captured by the digital camera; generate askin tone profile for pixels within the region of interest; and displaya predetermined makeup product recommendation based on the skin toneprofile.
 12. The system of claim 11, wherein the reference image isdepicted on one of the following objects: a white balance card, a colorchecker, a banknote, a credit card, photocopy paper, tissue paper, amobile phone, a non-glossy white object.
 13. The system of claim 11,wherein the parameters of the digital camera comprise at least one of:white balance level; exposure compensation; and gamma correction. 14.The system of claim 11, wherein the processor defines the region ofinterest in the facial region by: determining a color distance betweenpixels in the facial region and one or more predetermined target skintones; and designating pixels within a threshold color distance of theone or more predetermined target skin tones as part of the region ofinterest.
 15. The system of claim 11, wherein the processor defines theregion of interest in the facial region by: identifying locations ofpredetermined feature points within the facial region; and defining aboundary of the region of interest based on the locations of thepredetermined feature points.
 16. The system of claim 11, wherein theprocessor generates the skin tone profile for the pixels within theregion of interest by: generating a luminance histogram for the pixelswithin the region of interest; removing predetermined portions of theluminance histogram to generate a target histogram portion; determininga dominant color value based on the target histogram portion; andgenerating the skin tone profile based on the determined dominant colorvalue.
 17. The system of claim 11, wherein the processor generates theskin tone profile for the pixels within the region of interest by:extracting an illumination layer and a reflectance layer from the pixelswithin the region of interest; and generating the skin tone profilebased on the reflectance layer.
 18. A non-transitory computer-readablestorage medium storing instructions to be implemented by a computingdevice having a processor, wherein the instructions, when executed bythe processor, cause the computing device to at least: obtain areference image depicting at least one reference color; calibrateparameters of a digital camera based on the at least one referencecolor; capture, by the digital camera, a digital image of an individualutilizing the calibrated parameters; define a region of interest in afacial region of the individual depicted in the digital image capturedby the digital camera; and generate a skin tone profile for pixelswithin the region of interest; and display a predetermined makeupproduct recommendation based on the skin tone profile.
 19. Thenon-transitory computer-readable storage medium of claim 18, wherein thereference image is depicted on one of the following objects: a whitebalance card, a color checker, a banknote, a credit card, photocopypaper, tissue paper, a mobile phone, a non-glossy white object.
 20. Thenon-transitory computer-readable storage medium of claim 18, wherein theparameters of the digital camera comprise at least one of: white balancelevel; exposure compensation; and gamma correction.
 21. Thenon-transitory computer-readable storage medium of claim 18, wherein theprocessor defines the region of interest in the facial region by:determining a color distance between pixels in the facial region and oneor more predetermined target skin tones; and designating pixels within athreshold color distance of the one or more predetermined target skintones as part of the region of interest.
 22. The non-transitorycomputer-readable storage medium of claim 18, wherein the processordefines the region of interest in the facial region by: identifyinglocations of predetermined feature points within the facial region; anddefining a boundary of the region of interest based on the locations ofthe predetermined feature points.
 23. The non-transitorycomputer-readable storage medium of claim 18, wherein the processorgenerates the skin tone profile for the pixels within the region ofinterest by: generating a luminance histogram for the pixels within theregion of interest; removing predetermined portions of the luminancehistogram to generate a target histogram portion; determining a dominantcolor value based on the target histogram portion; and generating theskin tone profile based on the determined dominant color value.